An imaging sensor, an image processing device and an image processing method

ABSTRACT

An image processing device has circuitry, which is configured to obtain global image data including image data and spectral data, the global image data representing a global image area and to input the global image data to machine learning model for generating output spectral data, wherein the neural network is configured to transform the obtained image data into the output spectral data based on the obtained spectral data.

TECHNICAL FIELD

The present disclosure, generally pertains to an imaging sensor, an ageprocessing device and an image processing method.

TECHNICAL BACKGROUND

Generally, neural networks, such as Deep Neural Network (DNN) andConvolutional Neural Network (CNN) are known, and they are used in aplurality of technical fields, for example in image processing. Knownimage processing devices may use DNN and CNN for image reconstruction,multispatial and multispectral image transformation, multispatial andmultispectral image generation, object recognition and the like.

Moreover, DNN and CNN typically have an input layer, an output layer andmultiple hidden layers between the input layer and the output layer. Inimage processing, a neural network may be trained to output imageshaving high spectral resolution, using as an input to the neuralnetwork, a color channel image, such as an RGB image (having red, greenand blue color channels), captured by a color sensor, and a spectralchannel image, such as a multispectral or hyperspectral image, captureby multispectral sensor.

Although there exist techniques for image processing, it is generallydesirable to improve imaging sensors, image processing devices andmethods.

SUMMARY

According to a first aspect, the disclosure provides an image processingdevice comprising circuitry configured to obtain global image dataincluding image data and spectral data, the global image datarepresenting a global image area, to input the global image data to amachine learning model for generating output spectral data, wherein themachine learning model is configured to transform the obtained imagedata into the output spectral data based on the obtained spectral data.

According to a second aspect, the disclosure provides an imageprocessing method comprising obtaining global image data including imagedata and spectral data, the global image data representing a globalimage area, inputting the global image data to a machine learning modelfor generating output spectral data, wherein the machine learning modelis configured to transform the obtained image data into the outputspectral data based on the obtained spectral data.

Further aspects are set forth in the dependent claims, the followingdescription and the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are explained by way of example with, respect to theaccompanying drawings, in which:

FIG. 1 illustrates an exemplary embodiment of an imaging sensor;

FIG. 2 shows a block diagram of an embodiment of an image, processingsystem;

FIG. 3 shows an embodiment of a global image area, in which spectralinformation is distributed;

FIG. 4 shows an embodiment of a global image area, in which spectralinformation is distributed in clusters;

FIG. 5 shows an embodiment of spectral information being grouped in aregion of a global image area;

FIG. 6 visualizes the application of a Convolutional Neural Network;

FIG. 7 shows a block diagram of an embodiment of a learning system;

FIG. 8 illustrates an embodiment of a processing scheme of a learningmethod of a Convolutional Neural Network;

FIG. 9 illustrates an embodiment of a proposed system, which is trainedbased on a machine-learning model using the learning method of FIG. 8 ;and

FIG. 10 is a flowchart of an embodiment of an image processing method.

DETAILED DESCRIPTION OF EMBODIMENTS

Before a detailed description of the embodiments under reference of FIG.1 is given, general explanations are made.

As indicated in the outset, it is generally known that multispectralimaging systems and common Red Green Blue (RGB) imaging systems are usedto capture and analyze images having high spectral resolution and highspatial resolution, respectively. Typically, a multispectral imagingdevice provides higher resolved spectral information than a common RGBimaging system, which typically only provides color channel information,namely for the red, green and blue colors. A multispectral sensingdevice is usually more expensive than an RGB imaging device andtypically a multispectral sensing device compromises resolution,acquisition time and cost.

Generally, the spatial resolution of a mosaic-array multispectral sensormay be lower than the spatial resolution of a common RGB sensor.However, since the design costs of a common RGB sensor, usually, areless, than the costs of a multispectral sensor, most imaging systemsfocus on spatial resolution rather than spectral, resolution.

It is known that multispectral imaging systems performhyper/multispectral image data reconstruction on the basis of an RGBimage, using deep learning techniques, in order to benefit from bothspatial and spectral resolution information.

As mentioned in the outset, neural networks, such as Deep Neural Network(DNN) and Convolutional Neural Network (CNN) are known, and they havereached state-of-the-art level performance in many domains, such as ofimage processing, image reconstruction, multispatial and multispectralimage generation, language processing and the like. CNN is a part of DNNthat are usually applied to analyzing visual imagery.

In particular, CNN uses image classification algorithms for imagetransformation, multispatial and multispectral image generation, imageclassification, image and video recognition, natural language processingand the like.

As it is generally known, a CNN, may have an input layer and an outputlayer, as well as multiple hidden layers. The hidden layers of a CNNtypically have a number, of convolutional layers i.e. pooling layers,fully connected layers and the like. Each convolutional layer within aneural network usually has attributes, such as an input having shape(number of images)×(image width)×(image height)×(image depth), a numberof convolutional kernels, acting like a filter, whose width and heightare hyper-parameters, and whose depth must be typically equal to that ofthe image. The convolutional layers convolve the input and pass theirresult to the next layer.

In some cases, it may be suitable, that the Conventional CNN is trainedsuch as to reconstruct a hyper/multispectral image from an RGB imagecaptured by an RGB sensor and a hyperspectral image, captured byhyperspectral sensor. In such cases, the CNN may be trained to integratedata acquired from the different capturing systems and to perform analignment process and/or interpolation process of RGB image datarepresenting the captured RGB image to the hyper/multispectral imagedata representing the captured hyper/multispectral image.

However, it has been recognized that, for example, in particularsituations, multi-spectral imaging is desired, combining highresolution, and large number of acquisitions (depending on acquisitiontime) or quick acquisitions (depending on acquisition time). In suchcases, it has been recognized that a Conventional CNN may not besuitable, since the resolution of a hyper/multispectral image is higherthan the resolution of an RGB image, that is, a hyper/multispectralimage has a few hundred pixels more than an RGB image, and by usingalignment process and/or interpolation process of the two differentimages may result to information loss. Moreover, in such cases, twodistinct sensors are required, in order to capture the two differentimages.

Consequently, some embodiments pertain to an imaging sensor including aphotosensitive area, which includes an imaging part and a spectralimaging part.

The imaging sensor may be any high resolution/high speed RGB imagingsensor, or the like. The imaging sensor may be specifically designed forsensing light of different wavelength range depending on the filterapplied on its pixels, which may be arranged in pixel array in a sensorplane. At least one of the pixels of the sensor may be amulti/hyperspectral pixel having a sensitivity in a multi- orhyperspectral range. The image sensor may generate image data on thebasis of the wavelength range of an incoming light.

The photosensitive area may be an area, such as the, or corresponding tothe photosensitive sensor plane, formed by an array of pixels in someembodiments. The photosensitive area includes an imaging part and aspectral imaging part. The photosensitive area may be based on at leastone of the following: CMOS (Complementary Metal Oxide Semiconductor),CCD (Charge Coupled Device), SPAD (Single Photon Avalanche Diode), CAPD(Current Assisted Photodiode) technology or the like.

The imaging part may be a common color channel imaging part, such as acommon RGB (Red, Green, Blue) imaging part, having a plurality ofpixels, for sensing light of different wavelength ranges, namely of redlight, green, light and blue light, without limiting the presentdisclosure in that regard. The plurality of pixels of the imaging partmay sense any other light color (channels).

In particular, the color channel of a specific color, for example red,green, or blue, may include information of multiple spectral channelsthat corresponds to the wavelength range of red, green, or blue,respectively. That is, the color channels may be considered as anintegration of the corresponding (multiple) spectral channels located inthe wavelength range of the associated color channel.

The spectral imaging part may be a spectral channel imaging part having,for example, at least one imaging portion for sensing light of differentwavelength ranges or spectral ranges. In particular, the spectralimaging part may be specifically designed for sensing light of differentwavelength range or spectral range depending on the filter applied onthe at least one imaging portion. For example, different kind of colorfilters, spectral filters or the like may be applied on the spectralimaging part than the kind of color filters, spectral filters or thelike applied on the RGB pixels of the imaging part. Therefore, spectralpixels may be sensitive to different wavelength range or spectral rangethan the RGB pixels of the photosensitive area of the imaging sensor.

In some embodiments, the spectral imaging part may include a pluralityof spectral imaging portions, such as pixels. The plurality of spectralimaging portions may be distributed in the photosensitive area or atleast a part of the plurality of spectral imaging portions, may begrouped in a region of the photosensitive area or at least one clusterincluding a plurality of spectral imaging portions may be distributed inthe photosensitive area or the spectral imaging portions may be locatedin the photosensitive area with any combination of all the above. Eachof the pixels may be configured to detect light in a specific spectralrange of the overall to be detected multispectral or hyperspectralwavelength or spectral range.

The plurality of spectral imaging portions may be arbitrarilydistributed in the photosensitive area or may be distributed based on apredetermined pattern. In addition, the plurality of spectral imagingportions may be embedded in the photosensitive area together with theimaging, part (and/or the pixels of the imaging part).

For example, returning to FIG. 1 , an exemplary embodiment of an imagingsensor is illustrated, which includes an imaging part and a spectralimaging part, as discussed herein.

As mentioned above, an imaging sensor, such as imaging sensor 1 of FIG.1 has a photosensitive area 2, which in turn includes an imaging part 3and a spectral imaging part 4. The imaging part 3 is represented bywhite colored squares and the spectral imaging part 4 is represented bychess patterned squares. In this embodiment, the spectral imaging part 4has a plurality of spectral imaging portions, which are distributed inthe photosensitive area 2. The imaging part 3 and the spectral imagingpart 4 may each have a filter for filtering appropriately the wavelengthof an incoming light. For example, information is acquired from theimaging part 3, such as imaging information (e.g. red, green and bluecolor channel, information) and information is acquired from thespectral imaging part 3, such as spectral information. The informationacquired from the imaging part 3 and the spectral imaging part 4 isrepresented by image data and spectral data, respectively.

The imaging sensor 1 generates, for example, global image data,including image data and spectral data that represent imaginginformation and spectral information, respectively, which is describedunder the reference of FIGS. 3 to 5 further below.

Some embodiments pertain to an image processing device includingcircuitry configured to obtain global image data including image dataand spectral data, the global image data representing a global imagearea, and to input the global image data to a machine learning model forgenerating output spectral data, wherein the machine learning model isconfigured to transform the obtained image data into the output spectraldata based on the obtained spectral data.

The image processing device may be a digital (video) camera, asurveillance camera, a biometric device, a security camera, amedical/healthcare device, a remote sensing device, a food inspectiondevice, an edge computing enabled image sensor, such as smart sensorassociated with smart speaker, or the like, a motor vehicles device, asmartphone, a personal computer, a laptop computer, a personal computer,a wearable electronic device, electronic glasses, or the like, acircuitry, a processor, multiple processors, logic circuits or a mixtureof those parts.

The circuitry may include one or more processors, logical circuits,memory (read only memory, random memory, etc., storage memory, i.e. harddisc, compact disc, flash drive, etc), an interface for communicationvia a network, such as a wireless network, internet, local area network,or the like, a CMOS (Complementary Metal Oxide Semiconductor) imagesensor, a CCD (Charge Coupled-Device) image sensor, or the like, and itmay include the image sensor discussed herein.

The global image data may be generated by the image sensor, as mentionedabove. The global image data may be also obtained from a memory includedin the device, from an external memory, etc., from an artificial imagegenerator, created via computer generated graphics, or the like.

The image data may be Red Green Blue (RGB) image data. The image datamay be represented by a number of color channels, for example threecolor channels, such as Red, Green and Blue, or the like. The colorchannel of a specific color, for example red, green, or blue, mayinclude information of multiple spectral channels that corresponds tothe wavelength range of red, green, or blue, respectively. That is, thecolor channels may be considered as an integration of the corresponding(multiple) spectral channels located in the wavelength range of theassociated color channel. Moreover, the global image data represent aglobal image area, which, for example, formed by a plurality of pixels.

The spectral data may be represented by any number of spectral channels.The spectral data may be represented by multiple spectral channels, andmay be spectral data, such as multispectral data or hyperspectral data.

The above mention image processing device may be part of an imageprocessing system, such as a digital camera, an RGB camera, asurveillance camera, image acquisition systems, professional cameras,industrial equipment, or the like. An exemplary embodiment of an imageprocessing system, including an image processing device, is described inthe following under the reference of FIG. 2 .

FIG. 2 shows a block diagram of an embodiment of an image, processingsystem, which comprises an image processing device, as described herein.

An image processing system, such as the digital camera 11, has an imagesensor, such as the imaging sensor 1, described under the reference ofFIG. 1 above, an image processing device 12, and a display 13. In thepresent embodiment, the image processing device 12 includes a circuitry17 with an interface 19, a Central Processing Unit (CPU) 20, includingmultiple processors including Graphics Processing Units (GPUs), a memory21 that includes a RAM, a ROM and a storage memory and a trained CNN 22(which is stored in a memory).

The image processing device 12 acquires, through the interface 19, data,such as global image data 14, including image data, such as image data15 and spectral data, such as spectral data 16. The global image data 14represent an image of a target scene been captured with the digitalcamera 11. The image data 15 are represented by a number of colorchannels, namely Red, Green and Blue, in this embodiment. In addition,the image data 15 represent imaging information that corresponds toinformation acquired from the imaging part 3 of the photosensitive area2 of the imaging sensor 1 of FIG. 1 . The spectral data 16 aremultispectral image data, which are represented by a number of spectralchannels. The spectral data 16 represent spectral information, thatcorresponds to information acquired from the spectral imaging part 4 ofthe photosensitive area 2 of the imaging sensor 1 of FIG. 1 . In thisembodiment, the spectral information is acquired from a plurality ofspectral imaging portions grouped in the right upper region of thephotosensitive area 2 of the imaging sensor 1 of FIG. 1 , withoutlimiting the present disclosure in that regard.

The global image data 14, being represented by a number of colorchannels, are transmitted to the CPU 20, which inputs the global imagedata 14 into the CNN 22 for generating multispectral data, beingrepresented by a number of spectral channels. The CNN 22 has beentrained in advance to transform the obtained image data 15 into themultispectral data, such as output spectral data 18, based on theobtained spectral data 16. The output spectral data 18 represent anoutput image being displayed on the display 13.

The implementation of the above described image processing system 11,having the image sensor 1, may result to high-resolution multi-spectral,imaging by means of full resolution RGB (or similar) imaging andlow-resolution multi-spectral imaging. Hence, a multi/hyperspectraldevice may be designed having a low cost, high resolution, and highacquisition speed.

The CNN 22 may be a CNN trained from the scratch using amachine-learning algorithm or may be a CNN trained using a previouslytrained machine-learning algorithm.

As mentioned above, the global image data 14, including image data 15and spectral data 16, represent a global image area, which, for example,formed by a plurality of pixels. In some embodiments, the global imagearea may include more pixels represented by the obtained image data thanpixels represented by the obtained spectral data (i.e. the pixelsrepresenting spectral information may be sparse compared to the pixelsrepresenting imaging information (RGB information)). The obtainedspectral data may represent spectral information and the obtained imagedata may represent imaging information. In some embodiments, the imaginginformation may correspond to information acquired from an imaging partof a photosensitive area of an imaging sensor, such as the imaging part3 of the photosensitive area 2 of the imaging sensor 1 of FIG. 1 .Accordingly, the spectral information may correspond to informationacquired from a spectral imaging part of a photosensitive area of animaging sensor, such as the spectral imaging part 4 of thephotosensitive area 2 of the imaging sensor 1 of FIG. 1 .

Exemplary embodiments of global image data, including image data andspectral data that represent imaging information and spectralinformation, respectively, distributed in a global image area, asdescribed herein, are illustrated in FIGS. 3 to 5 in the following.

FIG. 3 shows an embodiment of global image data representing a globalimage area, in which spectral information, acquired from spectralimaging portions of an imaging sensor, is distributed. In particular, animaging sensor, such as the imaging sensor 1 generates global image data14, including image data 15 and spectral data 16, as discussed herein.The global image data 14 represent a global image area, such as image31. The image data 15 represent imaging information 32 and the spectraldata 16 represent spectral imaging information 33. The spectral imaginginformation 33 is acquired from a plurality of spectral imaging portionsdistributed in a photosensitive area of an imaging sensor, such thephotosensitive area 2 of the imaging sensor 1 of FIG. 1 . That is, fromeach of the plurality of spectral imaging portions, sub-spectral imaginginformation 34 is acquired. Therefore, the spectral imaging information33 is distributed in the global image area, in this embodiment.

An embodiment of spectral information, acquired from spectral imagingportions of an imaging sensor, being distributed in clusters in a globalimage area, is shown in FIG. 4 . As described under the reference ofFIG. 3 , an imaging sensor, such as the imaging sensor 1 generatesglobal image data 14, including image data 15 and spectral data 16. Theglobal image data 14 represent a global image area, such as image 31,in, which imaging information 32 and spectral imaging information 33, isdistributed. The spectral imaging information 33 is acquired from aplurality of spectral imaging portions grouped in clusters anddistributed in a photosensitive area of an imaging sensor, such thephotosensitive area 2 of the imaging sensor 1 of FIG. 1 Therefore, thespectral imaging information 33 is distributed in clusters 35 in theglobal image area, in this embodiment.

Another embodiment of spectral information, acquired from spectralimaging portions of an imaging sensor, being grouped in a region of aglobal image area, is illustrated in FIG. 5 . An imaging sensor, such asthe imaging sensor 1 generates global image data 14, including imagedata 15 and spectral data 16, as discussed herein. The spectral imaginginformation 33, represented by spectral data, is acquired from aplurality of spectral, imaging portions grouped in a region of aphotosensitive area of an imaging sensor, such the photosensitive area 2of the imaging sensor 1 of FIG. 1 Therefore, the spectral imaginginformation 33 is grouped in a region 36 of the right upper corner ofthe global image area, in this embodiment.

The embodiments described above under the reference of FIGS. 3 to 5 donot limit the present disclosure in that regard. The spectral imaginginformation may be distributed in the global image area, with anycombination of the above mentioned distribution methods.

In some embodiments, the machine learning model may be a neural networkand in particular, the machine learning model may be a convolutionalneural network (CNN), without limiting the present disclosure in thatregard. For example, in some embodiments, the convolutional neuralnetwork may include convolutional layers, or may also include local orglobal pooling layers, such as max-pooling layers, which reduce thedimensions of the image data, as it is generally known. The poolinglayers may be used for pooling, which is a form of non-lineardown-sampling, such as spatial pooling, namely max-pooling, averagepooling, sum pooling, or the like.

The generation of the output spectral data may be performed eitherduring a training phase of a neural network, such as a CNN, or may be ageneration of the output spectral data with an already trained neuralnetwork, such as a trained CNN, for example, for extracting informationfrom the image data (e.g. object recognition, or recognition of otherinformation in the image data, such as spatial information, spectralinformation, patterns, colors, etc). Hence, the neural network may be atrained neural network or an un-trained neural network, wherein theun-trained neural network may be trained on-the-fly, e.g. duringoperation of the associated (image processing) device.

Moreover, the neural network may be part of the image processing device,e.g. stored in a storage or memory of the image processing device, orthe image processing device may have access to a neural network, e.g.based on inter-processor communication, electronic bus, network(including internet), etc.

The general principle of the usage of the CNN is exemplary illustratedin FIG. 6 , which shows generally in the first line the CNN structure,and in the second line the basic principle of building blocks. Theprinciples of a CNN and its application in imaging is generally knownand, thus, it is only briefly discussed in the following under referenceof FIG. 6 .

The input image includes for example three maps or layers (exemplaryred, green and blue (RGB) color information) and N times N blocks. TheCNN has a convolutional layer and a subsequent pooling layer, whereinthis structure can be repeated as also shown in FIG. 6 . Theconvolutional layer includes the neurons. By applying a kernel (filter)(see convolution kernels in the second line) on the input image, arespective feature map can be obtained. The pooling layer, which isbased in the present embodiment on the Max-Pooling (see second line,“Max-Pooling), takes the information of the most active neurons of theconvolution layer and discards the other information. After severalrepetitions (three in FIG. 6 ), the process ends with a fully-connectedlayer, which is also referred to as affine layer. The last layerincludes typically a number of neurons, which corresponds to the numberof object classes (output features) which are to be differentiated bythe CNN. The output is illustrated in FIG. 6 , first line, as an outputdistribution, wherein the distribution is shown by a row of columns,wherein each column represents a class and the height of the columnrepresents the weight of the object class. The different classescorrespond to the output or image attribute features, which are outputby the CNN. The classes are, for example, “people, car, etc.” Typicallyseveral hundred or several thousand of classes can be used, e.g. alsofor object recognition of different objects.

In some embodiments, the convolutional neural network (CNN) may betrained based on the obtained image data and the obtained spectral data.Thus, in some embodiments, the convolutional neural network may betrained to transform the obtained image data into the output spectraldata based on spectral information acquired from a plurality of spectralimaging portions of a photosensitive area of an imaging sensor.

As discussed herein, the CNN may be a trained CNN or, may be anuntrained CNN. The training options of a CNN, as discussed above, isdescribed under the reference of FIG. 7 in the following.

An embodiment of a learning system 40, shown as a block diagram, isillustrated in FIG. 7 that generates a machine-learning model based onwhich a neural network, such as CNN 22 is trained.

The learning system 40 includes data acquisition 41, data set 42, atraining 43 a from the scratch of a system, such as machine learningalgorithm, or a training 43 b of a pre-trained system, such as amachine-learning algorithm, based on a pre-trained model 44, and amachine-learning model 45 a and 45 b.

Global image data 14 including image data 15 and spectral data 16,representing a number of images, for example, a hundred to a thousand(100-1000) images, are acquired during data acquisition 41, and thus,the data set 42 is generated and stored into the memory 21 of the imageprocessing device 12. The number of a hundred to a thousand (100-1000)images is a relatively large number of images, which results in enoughdata to train a machine-learning algorithm. The machine-learningalgorithm may be trained with a training 43 a, such as a training fromthe scratch. The CNN 22 may be trained with the machine-learningalgorithm, such as to use, as ground truth, the spectral information 33,acquired from the spectral imaging portions included in the spectralimaging part 4 of the imaging sensor 1 of FIG. 1 , having for example, aplurality of multi/hyperspectral pixels. The CNN 22 is further trainedto transform the imaging information 32, acquired from the imaging part3 of the imaging sensor 1 of FIG. 1 , having for example Red, Green,Blue (RGB) pixels, to multi/hyperspectral information, such as theinformation represented by the output spectral data 18 of FIG. 2 , basedon the spectral data 16 representing spectral information 33. Thelearning system 40 generates a learned model 45 a, which is stored intothe memory 21 of the image processing device 12.

Alternatively, a training 43 b of a pre-trained system may be used toperform the same transformation, requiring fewer acquisitions. Thepre-trained system may be trained based on a pre-trained model 44. Thelearning system 40 generates a learned model 45 b, which is stored intothe memory 21 of the image processing device 12.

The training process may be, realized in the image processing system 11,such as a camera, in a cloud service, in a user computer, in a dedicateddevice, or the like.

Some embodiments pertain to an image processing method, which may beperformed by the image processing device described herein, or any otherelectronic device, processor, or other computing means or the like. Themethod includes obtaining global image data including image data andspectral data, the global image data representing a global image area,and inputting the global image data to a machine learning model forgenerating output spectral data, wherein the machine learning model isconfigured to transform the obtained image data into the output spectraldata based on the obtained spectral data.

As mentioned, the global image area may include more pixels representedby the obtained image data than pixels represented by the obtainedspectral data. In addition, the obtained spectral data may representspectral information and the obtained image data may represent imaginginformation. The spectral information may correspond to informationacquired from a spectral imaging part of a photosensitive area of animaging sensor. Moreover, the machine learning model a neural networkand, in particular, the machine learning model may be a convolutionalneural network, which may be trained based on the obtained image dataand the obtained spectral data. Furthermore, the convolutional neuralnetwork may be trained to transform the obtained image data into theoutput spectral data based on spectral information acquired from aplurality of spectral imaging portions of a photosensitive area of animaging sensor. The convolutional neural network may be trained based ona learning algorithm, the learning algorithm computing the loss functioncorresponding to the image data and the loss function corresponding tothe spectral data.

Referring to FIG. 8 , an embodiment of a processing scheme of a learningmethod 50 of the CNN 22 is illustrated, in which, the obtained imagedata 15 are transformed into the output spectral data 18 based on theobtained spectral data 16.

The image processing device 12 inputs (arrow 56) into the CNN 22 theglobal image data 14, which include the image data 15, such as RGB imagedata and the spectral data 16, such as multispectral/hyperspectral data.As discussed herein, the global image data 14 represent a global imagearea, in which the number of pixels represented by the image, data 15 islarger than the number of pixels represented by the spectral data 16.Thus, the global image data 14 represent an RGB image having sparsemultispectral/hyperspectral information. The CNN 22 predicts spectraldata (arrow 57), such as multispectral/hyperspectral data, and outputsthis as output spectral data 18 (e.g. after the following learningprocess). The predicted spectral data are converted (arrow 58) back toimage data 52 using a back transformation 51. The back transformation 51is performed, using a physical model, and, in particular, by integratingthe spectrum according to certain color sensitivities (red, green,blue). An RGB loss function 53 indicating a loss in the RGB domain iscomputed, based on the image data 15 and the back-transformed image data52, which then is fed to the CNN 22. Accordingly, a spectral lossfunction 54 indicating a loss in the spectral domain is computed, basedon the spectral data 16 and the predicted spectral data, which then isfed to the CNN 22. The spectral loss function 54 may be, for example, aone-to-one comparison of predicted spectral data, with the closest (e.g.according to RGB values) multispectral/hyperspectral data available fromthe sparse spectral information. In addition, a smoothening 55 isperformed on the predicted spectral data, for example, a total variationminimization. The RGB loss function 53 and the spectral loss function 54may be used consecutively, for example by first training a convolutionalneural network by using the loss in the RGB domain, such as the RGB lossfunction 53, with a smoothness constraint of the smoothening 55, andthen fine-tuning the learning model 45 a, 45 b by using the sparsemultispectral/hyperspectral information.

An embodiment of a proposed system 60, in which the image processingsystem 11 including the imaging sensor 1, the image processing device 12and the display 13, is trained based on the machine-learning model 45 a,45 b, using the learning method 50, is illustrated in FIG. 9 as a blockdiagram.

As mentioned above, the imaging sensor 1 is a high resolution/high speedRGB sensor, which acquires global image data 14 including image data 15representing high-resolution RGB images and spectral data 16representing low-resolution multi/hyperspectral images. The global imagedata 14 are input into the image processing device 12 for generatingoutput spectral data 18 representing multi/hyperspectral images, whichmay be displayed on the display 13. The output spectral data 18 aregenerated by the CNN 22, which is trained based on the machine-learningmodel 45 (45 a or 45 b), using the learning method 50. Using thelearning method 50, the image data 15 are transformed into the outputspectral data 18 based on the spectral data 16, as discussed herein.

In the following, an image processing method 70, which is performed bythe image processing device 12 and/or the image processing system 11 insome embodiments, is discussed under reference of FIG. 10 .

At 71, global image data, such as global image data 14, are obtained bythe image processing device 12 and/or the image processing system 40, asdiscussed above.

The global image data may be obtained from an image sensor, such as theimaging sensor 1, or from a memory included in the device, from anexternal memory, etc., or from an artificial image generator, createdvia computer generated graphics, or the like.

The global image data, at 72, are input into a convolutional, neuralnetwork, such as CNN 22, for generating, at 73, output multispectral(MS) data, such as the output spectral data 18, as discussed above.

The global image data includes image data that may be represented by anumber of color channels, such as Red, Green and Blue, or the like andspectral data.

The global image data represent a global image area, which includes morepixels represented by the obtained image data than the pixelsrepresented by the spectral data, as discussed above.

At 74, the convolutional neural network transforms the obtained imagedata into the output spectral data based on the obtained spectral data.

The image data represent imaging information and the obtained spectraldata represent spectral information, which corresponds to informationacquired from a spectral imaging part of a photosensitive area of animaging sensor.

At 75, the generated output spectral image data are output.

It should be recognized that the embodiments describe methods with anexemplary ordering of method steps. The specific ordering of methodsteps is however given for illustrative purposes only and should not beconstrued as binding.

The method as described herein is also implemented in some embodimentsas a computer program causing a computer and/or a processor to performthe method, when being carried out on the computer and/or processor. Insome embodiments, also a non-transitory computer-readable recordingmedium is provided that stores therein a computer program product,which, when executed by a processor, such as the processor describedabove, causes the methods described herein to be performed.

All units and entities described in this specification and claimed inthe appended claims can, if not stated, otherwise, be implemented asintegrated circuit logic, for example on a chip, and functionalityprovided by such units and entities can, if not stated otherwise, beimplemented by software.

In so far as the embodiments of the disclosure described above areimplemented, at least in part, using software-controlled data processingapparatus, it will be appreciated that a computer program providing suchsoftware control and a transmission, storage or other medium by whichsuch a computer program is provided are envisaged as aspects of thepresent disclosure.

Note that the present technology can also be configured as describedbelow.

(1) An imaging sensor comprising:

-   -   a photosensitive area including an imaging part and a spectral        imaging part.

(2) The imaging sensor of (1), wherein the spectral imaging partincludes a plurality of spectral imaging portions.

(3) The imaging sensor of (2), wherein the plurality of spectral imagingportions are distributed in the photosensitive area.

(4) The imaging sensor of (2), wherein at least a part of the pluralityof spectral imaging portions are grouped in a region of thephotosensitive area.

(5) An image processing device comprising circuitry configured to:

-   -   obtain global image data including image data and spectral data,        the global image data representing a global image area;    -   input the global image data to a neural network for generating        output spectral data, wherein the neural network is configured        to transform the obtained image data into the output spectral        data based on the obtained spectral data.

(6) The image processing device of (5), wherein the global image areaincludes more pixels represented by the obtained image data than pixels,represented by the obtained spectral data.

(7) The image processing device of (5) or (6), wherein the obtainedspectral data represent spectral information and the obtained image datarepresent imaging information.

(8) The image processing device of (7), wherein the spectral informationcorresponds to information acquired from a spectral imaging part of aphotosensitive area of an imaging sensor.

(9) The image processing device of anyone of (5) to (8), wherein theneural network is a convolutional neural network.

(10) The image processing device of (9), wherein the convolutionalneural network is trained based on the obtained image data and theobtained spectral data.

(11) The image processing device of (9), wherein the convolutionalneural network is trained to transform the obtained image data into theoutput spectral data based on spectral information acquired from aplurality of spectral imaging portions of a photosensitive area of animaging sensor.

(12) The image processing device of anyone of (5) to (11), wherein theobtained image data are RGB image data and the obtained spectral dataare multispectral or hyperspectral data.

(13) An image processing method comprising:

-   -   obtaining global image data including image data and spectral        data, the global image data representing a global image area;    -   inputting the global image data to a neural network for        generating output spectral data, wherein the neural network is        configured to transform the obtained image data into the output        spectral data based on the obtained spectral data.

(14) The image processing method of (13), wherein the global image areaincludes more pixels represented by the obtained image data than pixelsrepresented by the obtained spectral data.

(15) The image processing method of (13) or (14), wherein the obtainedspectral data represent spectral information and the obtained image datarepresent imaging information.

(16) The image processing method of (15), wherein the spectralinformation corresponds to formation acquired from a spectral imagingpart of a photosensitive area of an imaging sensor.

(17) The image processing method of anyone of (5) to (16), wherein theneural network is a convolutional neural network.

(18) The image processing method of (17), wherein the convolutionalneural network is trained based on the obtained image data and theobtained spectral data.

(19) The image processing method of (17), wherein the convolutionalneural network is trained to transform the obtained image data into theoutput spectral data based on spectral information acquired from aplurality of spectral imaging portions of a photosensitive area of animaging sensor.

(20) The image processing method of (17), wherein the convolutionalneural network is trained based on a learning algorithm, the learningalgorithm computing the loss function corresponding to the image dataand the loss function corresponding to the spectral data.

(21) A computer program comprising program code causing a computer toperform the method according to anyone of (13) to (20), when beingcarried out on a computer.

(22) A non-transitory computer-readable recording medium that storestherein a computer program product, which, when executed by a processor,causes the method according to anyone of (13) to (20) to be performed.

1. An image processing device comprising circuitry configured to: obtainglobal image data including image data and spectral data, the globalimage data representing a global image area; input the global image datato a machine learning model for generating output spectral data, whereinthe neural network is configured to transform the obtained image datainto the output spectral data based on the obtained spectral data. 2.The image processing device of claim 1, wherein the global image areaincludes more pixels represented by the obtained image data than pixelsrepresented by the obtained spectral data.
 3. The image processingdevice of claim 1, wherein the obtained spectral data represent spectralinformation and the obtained image data represent imaging information.4. The image processing device of claim 3, wherein the spectralinformation corresponds to information acquired from a spectral imagingpart of a photosensitive area of an imaging sensor.
 5. The imageprocessing device of claim 1, wherein the machine learning model is aconvolutional neural network.
 6. The image processing device of claim 5,wherein the convolutional neural network is trained based on theobtained image data and the obtained spectral data.
 7. The imageprocessing device of claim 5, wherein the convolutional neural networkis trained to transform the obtained image data into the output spectraldata based on spectral information acquired from a plurality of spectralimaging portions of a photosensitive area of an imaging sensor.
 8. Theimage processing device of claim 1, wherein the obtained image data areRGB image data and the obtained spectral data are multispectral orhyperspectral data.
 9. An image processing method comprising: obtainingglobal image data including image data and spectral data, the globalimage data representing a global image area; inputting the global imagedata to a machine learning model for generating output spectral data,wherein the machine learning model is configured to transform theobtained image data into the output spectral data based on the obtainedspectral data.
 10. The image processing method of claim 9, wherein theglobal image area includes more pixels represented by the obtained imagedata than pixels represented by the obtained spectral data.
 11. Theimage processing method of claim 9, wherein the obtained spectral datarepresent spectral information and the obtained image data representimaging information.
 12. The image processing method of claim 11,wherein the spectral information corresponds to information acquiredfrom a spectral imaging part of a photosensitive area of an imagingsensor.
 13. The image processing method of claim 9, wherein the machinelearning model is a convolutional neural network.
 14. The imageprocessing method of claim 13, wherein the convolutional neural networkis trained based on the obtained image data and the obtained spectraldata.
 15. The image processing method of claim 13, wherein theconvolutional neural network is trained to transform the obtained imagedata into the output spectral data based on spectral informationacquired from a plurality of spectral imaging portions of aphotosensitive area of an imaging sensor.
 16. The image processingmethod of claim 13, wherein the convolutional neural network is trainedbased on a learning algorithm, the learning algorithm computing the lossfunction corresponding to the image data and the loss functioncorresponding to the spectral data.